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Caution indices of unusual response patterns for individual test 
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*r' INTRODU'CTIQN 



Millions of achievement tests are admiftis.tered eacl^year ^tS" ptuaents 
in this country. The types of tests are quite varied as are the purposes 
of the testing. Tests are often used, as in thd case of .4 the National . <i 

Assessment of Educational Progress and several statewide testing programs, 
"to provide a general assessment of "large aggregates pf students. Survey 
tests are often used to morfltor achievement of students at the school or 
district level. Individual student achievement Is 'also assessed for many* 
puirposes. To name a few, test result.5 are used to report achievement to 
parents, for diagnosis of learning difficulties, for grade-to-grade pro- 
motion and for the award of high school diplomas. Mandated test use in 
program evaluations and the selection of students for special programs 
accounts for "another large segment of test use. 

In most uses -of achievement tests, whether with individual pupils 
or with aggregates, the primary focus is on a few summary scojes that 
are based on the number of items answered correctly. The focus on the 
number of correct an[«wer& or some transformation of It (e.g., a grade- 
equivalent score) is natural. Certainly, the overall level of performance 
- in a content area such a^ arithmetic is a major consideration. Further- 
more, the interrelatedness of subsets of achievement 1:est items (e.g., 
addition- and subtraction items) makes it difficult- to abstract" reliable 
information about special strengths and weaknesses for individuals or for 
groups of students in addition to a global summary score. 

Nonetheless, it is clear that the same number-right score can be 
obtained in many different ways. Even on a short test of just five items, 
a score of three right answers can be obtained by ten different combina- 
tions of correct and 'incorrect responses- The nutaber offtpossible combin- 
ations of right and wrong answers expands rapidly with increases in the 
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tesC length. On a twenty-l,tem test ."^ for example, there are 18A,75^if- 
ferent re3p4i?8> patterns 'that yield the same number right score of ten. 

Although the 18A,756'distinct patterns of response can be identified 
in the above example] it obviously is not feasible to provide differfent 
interpretations for each unique pattern. Responses to individual items 
are too unreliable. 'Many of the. possible patterns wil-V- not be encountered 
in practice aifd the frequeiicy of occurrence of any particular pattern will 
usually be too small to support' any clear general izatHTOns "beyond those 
that can be made from the simple number-right acore. 

The difficulties in using Response patterns to obtain diagnostic in- ^ 
formatiop not contained in the number-right score explain the relatively 
limited use that has been made of such information. Recently, however, 
some new arid' potential-ly more powerful techniques have been developed for 
identifying atypical response patterns. The introduction of these tech- 
niqir|s has led to a renewed interest in using information contained in 
response patterns to identify individua-^ with unusual patterns (see, for 
example, Harnisch & Linn, 1981,; Levine i Rubin, "''igyg; Sato,, 1975, 1981; 

t. 

Tatsuoka & tatsuoka,-1980; der Fliel:, 1977;). 

The identification of utbusual response patterns for individuals has 
several possible uses. The most obvious use is simply to identify students 
for whom special caution Is needed in i;iterpretin|^ their total correct 
scores. One of thd purposes of this study is to explore the use of an 
index of the degree to which an, individual ' s response pattern is unusual 
and whether ^his index is related to background characteristics of the 

♦ 

individual test takers. 

The identification ^of persons foi? whom special caution is required 
in interpreting* total scores is important. Potentially even more" impor- 
tant, however, is the possible diagnostic value of the response pattern. 



For example/ the content characteristics of items that contribute to a 
high caution index for. a student may identify particular strengths or 
gaps in the student's knowledge. 

The role of specific test content of ten . receives relativelyMittle 
attention when the ranking of students on a global dimension is empha- 
sized. For some purposes, it is useful, for-example,/to know that 
students in an instructional program have an average score well below 
the national mean of a fourth-grade mathematics test. But, by itself, 
this statement is uninformative about the types of arithmetic skills the 
8tudent%have. As shown by Porter, Schmidt , Floden, and Freeman (1978), 

there is wide variation in content coverage of the fourth-grade matheraa- 

V ' " 

tics tests from the four most widely used achiev,ement test batteries. 
Furthermore, the level of performance can be expected to be influenced 
by the degree of overlap between the instructional content and the 
content of the test (e.g., Hadaus, Kellaghan, Rakow, & King, 1979). 
Indeed, the specific match between the format of instructional exercises 
and the format of test questions can have a substanLflal impact on test 
scores (Alderman, Swintoq, & Braswell, 1979; House, , Glass, McClean, & 
Walker, 1978).' ' 

To the degree that content coverage is important and that schools 
vary in their content coveri^ge, then schools would be exT>ected to vary 
not only in their overall performance as measured by the total test score, 
but in the relative difficulty of particular subsets of items. The second 
major purpose of this study was to determine if schools could be reliably 
distinguished in terras of consistent response patterns that deviated from 
the national norm. The deraonstration of systematic between school dif- 
fereuces in response patterns and the association of these differences 
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with differences in instructional practices has potentially important 
implications for test use in evaluation studies. The match between tests 
and content coverage is an important consideration in evaluation studies, 
but one that has frequently been given little attention. The indices 
that were used in this study provide a potential means of identifying 
situations where comparisons between groups solely on the basis of total 
scores may be misleading. The clusters of items tha,t contribute to high 
caution indices provides alternate comparisons, and possibly more meaning- 
ful interpretations of observed performance outcomes. ^ 
^ REVIEW OF RESEARCH 

Two major types of appropriateness indices based upon the pattern of 
item responses to individual iteias have been defined. First, there are 
the indices -which are based upon item response theory (IRT) as described 
by Levine and Rubin (1979) and modifications of these indices as suggested 
•by Drasgow (1978). The x test of person fit that is sometimes used with 
applications of the Rasch model (e.g.", Wright, 1977) ,1s another example of 
an IRT based index. Second, there are the indices which are based directly 
uppon the pattern of right and wrong answers, such as the "caution" index 
proposed by Sato (1975), the modified caution index by Harnisch and Linn 
(1981), the U'' index by van der Fli'er (1977), the personal biserial by 
Donlon and Fischer, (1968) , the norm-conformity index by Tatsuoka and 
' Tatsuoka (1980), and the agreement and -disagreement indices discussed by 
Kane and Brennan (1980). 

The focus oi^ this final report, is on two activities conducted with 
the latte'^r type of indices, Sato's caution index and ^he modified caution 
index,- which do not require the use of IRT. Our primary purposes are 
(1) to evaluate the distributional and relational properties of the caution 
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Indices with the total test score, (2) to identify student background 
characteristics that are associated with unusual response patterns (i.e., 
with high values of C^, or C^*, and (3) to identify school characteris- 
tics that are associated with high incidence of aberrant response patterns. 
We will review briefly the caution indices and their algebraic formulation. 
We will then discuss the empirical interrelationship among the caution in- 
dices and their relationships with total number-right scores for data col- 
lected by National Assessment of Educational Progress (NAEP) from 13-year- 
olds in 1978 on two mathf-matifcs test packages. We will then discuss the 
student and school characteristics associated with high modified caution 
indices. Finally, we will examine the school and census division dif- 
ferences on the modified caution index, and content differences in items 
which contribute to school-to-school differences in the index. 

DEFINITION AND COMPARISON OF INDICES 
For purposes of defining and comparing indices of the degree to which 
an individual's response pattern is unusual It is convenient to start with 
a consideration of a matrix of zeros and ones- A row of the matrix is asso- 
ciated with each examinee and a column with each item. Ones are recorded 
for- correct responses and zeros for incorrect response. Rows and columns 
of the data matrix are permuted so that the , items (colurans) are arranged, 
from left to right in ascendin| order of difficulty, and examinees (rows) . 
arc arranged, from top to bottom in descending order of total number of 
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t answers. The resulting matrix has been called an "5-P" Table 



(student-problem table) by Sato (1975). (Seei Tatsuoka [Note 1] for a 
description in English.) 

If the items formed a perfect GuCtraan Scale (Guttraan, 1941), the S-P 
Table would consist of a section with all ones in the upper left-hand 
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coraer and all zeros in the lower-right-hand corner, A single step-like 
boundar/ line would separate the ones and the zeroes. In other words, 
anyone who responded correctly to a difficult item would also answer all 
easier items correctly. There would be no unusual response pattern^Ln 
the sense that is used for the indices described in this manuscript be- 
cause everyone who had a given total score would have the same pattern 
of responses* Of course, with responses to achievement test items, 
perfect Guttman scales cannot be expected* Consequently, the S-P Table 
, will be characterized by a predominance of ones in the upper-left hand 
corner and zeroes in the lower right-hand corner, but there will be many 
exceptions to the pattern, i .e . , ones in the region where mostly zeroes 
are found and vice versa* 

A small hypothetical ex.ample of an S-P Table with 18 examinees and 
5 items is shown in Table 1, The solid and dashed lines in Table 1 are 
known as the S-curve and P-curve respectively* Th^e S-curve (solid line) 
is obtained by drawing a vertical line for each row that has n^. items 
(columns) to the left of it where n^* is the total number of correct 
responses for the 1 — examinee. The P-curve (dashed line) is obtained 
by drawing a horizontal line in each column such that there are n.^ ex- 
aminees who answer item j correctly, 

Xnsert Table 1 about here 



For an ideal, or Guttraan-scalable S-P Table, the 3- and P-curves would 
coincide. The degree of divergence iadicateg^heterogeneity, degree of con- 
vergence Indicates homogeneity of the response patterns. Sato (1975) has 
developed an index based on the area between the S- and P-curves which is 



potentially useful In evaluating the homogeneity of the test (see Tatsuoka, 
Note 3). Of greater interest for our present purposes, however, is Sato's 
"caution" index. 

Sato's caution index, for the i— examinee, may bd defined as 
follows : 



= - J 



(I) 



n . 
1* 



V7here i = 1, 2, ... I, indexes the examinee, 

j = 1, 2, ... J, indexes the itsm, 

i if examinee i ansx^ers item j incorrectly, 

0 if examinee i answers item j incorrectly. 



"i, 



'^ij 



total correct for the i^— examinee, and 

t h 

total number of correct responses to the j — item. 



A parallel index for the j — item may be defined by simply reversing 
the roles of i and j in the above equation, but only the person index will 
be considered in the present paper. Values of C^, and for all of the 
indices described below, are listed in Table I. 

The name of the index comes from the notion" that a large value Is 
associated with examinees that have unusual response patterns. It denotes 
that some caution may be needed in interpreting a total correct score for 
an examinee. An unusual response pattern may result from guesslag, care- 
lessness, high arixiety, an unusual instructional history or other experien- 
tial background, a localized misunderstanding that influences responses to 
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a subset of items, or copying a neighbor's answers to certain questions. 
The key point is that the caution index provides information about an 
examinee that is not contained in the total score. A large value of the 
caution index raises doubts about the validity of the usual interpreta- 
tlon of the total score for an Individual. 

A modified form of Sato's caution Index, C^*, was introduced to 
yield a lower bound of 0 and an upper bound of 1. This modified caution 
Index, Cj^*, for the 1— examinee may be defined as follows: 



n. J 

- +1 -J 



y (1 - u..)n ; - 

. * _ l± . (2) 

^i ~ n. , 



Y n ; - y n . 

3=1 J j=J+l-n.. ^ 
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By being bound between 0 and 1, the modified caution index eliminates 
the extreme scores that are sometimes obtained on the caution index, 
especially in cases where a very high scoring examinee misses a single 
very easy item. This is seen as a potential advantage of C^*. 

Sato' s» caution index and the modified caution index have been briefly 
described. Both indices provide an indication of the degree to which an 
individual's response pattern departs from a norm. 

* DATA SOURCE ANT) PROCEDURE 
The data used in this investlgatioa carae from Booklet 4 and 5 given 
to thirteen-year olds during the 1977-1978 NAEP mathematics survey. 
Booklets 4 and 5 were selected because they contained the affective domain 
items related to attitudes toward mathematics along with the cognitive 
items related to math achievement. Public use data files were purchased 
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from the Education Commissioa of the States (ECS) CNote 2). Individual 
responses of 9-, 13-, 17-year olds to background, affective domain, ^and 
cognitive items are recorded for numerous booklets used at each respective 
age in the national survey. 
Sampliag 

A deeply stratified, multi-stage sampling design with oversampling 
of low-income and rural areas was used by NAEP (National Center for Edu- 
cational Statistics, 1978; Moore, Chromy, & Rogers, 197A). In the first 
stage, the United States is divided into geographical uri^its, including 
Standard Metropolitan Statistical Areas and counties or g^^roups of con- 
tiguous counties- These are stratified according to region and size of 
community. In the second stage, schools are randomly sampled within 
selected first-stage geographical units. Finally, from 10 to^ 35 age- 
eligible students are randomly selected within schools for testing. 

The NAEP test materials are organized into booklets sulta'ble for 
administration in a single sitting. Each booklet or "test package" 
contains a collection of items taken by a selected student. In the 
1978 mathematics assessment, 11 separate booklets were each administered 
to large national samples of 13-year-old students. 
Instruments 

Booklet 4. Affective items in Booklet 4 asked students questions 
about how much they liked mathematics and how useful they believed it was. 
The list of affective items Is given in Exhibit 1, used a Likert scale and 
were scored 1^ to 2* Affective items were recoded so that a value of 5 
would indicate a greater liking of math or a greater usefulness of mathe- 
matics.' One additional non-achievement test item scored "yes," "no" or 
"I don't know" was used from this booklet. The question asked of students 
was: Do you or your family own a hand calculator? 



items from the prlaclpalVs ques^l^nalre_lncluded In this study were 
Che school size, type of school (public versus private Catholic), percent 
of students black, number of students qualified to receive Title I and 
the percentage of students attending school whose parents are professional 
personnel, skilled workers, unskilled workers, farm workers, not employed, 

and on welfare. ^ , 

Cognitive items from Booklet 4 were categorized in terms of their 
content and format into four categories: arithmetic, geometry, tables 
and graphs, and miscellaneous. The listing of the cognitive item numbers 

into the tr^ res peet-ive aa-t-egor-i-es^i-s -gl-ven„i.n^Appendix- A . i — 

Booklet 5.' Affective items in Booklet 5 consisted of statements to 
which respondents indicated the easiness, importance and the degree to 
which they liked mathematics. Exhibit 2 provides a listing of affective 
domain items given in Booklet 5 which used a Likert scale and were scored 
1 to 5. The items were recoded so that a high value would indicate a 
greater amount of the domain being tapped. A similar additional affective 
item was used from Booklet 5 as well as the items from the principal's 
questionnaire that were used from Booklet 4. 

The cognitive items from Booklet 5 were similarly categorized into the 
same four categories as Booklet 4. A listing of the cognitive item numbers 
and respective item category ts given in Appendix B. 
Reliability of Affective Domain Scales 

The affective items listed in Exhibit 1 were classified into the Like 
Mathematics scale and the Usefulness of Mathematics scale . Reliability 
analyses of these scales were conducted yielding a Cronbach coefficient 
alpha of -49 for the Like scale and an alpha of .67 for the Usefulness 
scale . 
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The affective Items listed In Exhibit 2 were classified Into either 

the Importance, Easiness- -and-Like- SGales-^1 J8.ellabJL ULltLy: a na l y s e s^^f s e 

scales were conducted yielding 'a CronbacH alpha of •?! f6r the Importance 
scale, •66 for the Easiness scale and .69 for the Like scale. 

Each of these scales along with the student backgound arid school 
variables were used for the correlational analysis with the Individual 
caution Indices computed on all cognitive Items,, and on subsets of arith- 
metic and geometry Items. 
Data for Secondary Analysis 

The data tape provided by th e NAEP contains Information on 2,437 and 
2, A62 '13-year-olds responding to questions on Booklet A and Booklet 5, 
respectively. Each sample Is composed of an almost equal number of boys 
and girls, 50.8% boys In Booklet 4 and 48.5% boys In Booklet 5 • Most 
students were attending a public school (87.2% — Booklet 4; 89.9% — 
Booklet 5) as compared to a private school. The majority of the students 
are white while blacks and other racial groups are represented. Most 
parent-s of students In each sample were blue-collar workers, clerical or 
skills workers, somewhat fewer were classified as professional, managerial 
or farm workers. For eagh of the booklet samples, parents of approximately 
12% of the sample were on welfare while approximately another 7% were not 
regularly employed. ^ 
Data Analysis 

Computer programs were written in FORTRAN IV to compute the individual 
atid item caution Indices. Statistical programs from Statistical Package 
for the Social Sciences (SPSS) (Nie et al., 1975), Statistical Analysis 
System (SAS) (3AS User's Guide, 1979 Edition), and Finn's MULTIVARIANCE 
were used to answer ^he research questions. 
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TheNAEP saapling procedures requira that individual weighting pro- 
cedures be used to reflect the representative target population. In order 



to obtain imbiased estimates of population means we took the iTidivldual- 

weight value supplied by NAEP otr the tape file and created a weighted data 
file wltH observations being weighted in inverse proportion To their proba- 
bilities of being included in the saciple. All correlational analyses were 
based on the weighted data file for the sampling design, and therefore the 
results apply to a representative population of students^in the United 
States (Moore et al., 1974). For school analyses, a weighted procedure 
was followed as outlined by David Wright (1981, p. 30) which would assure 
the representativeness of our nations' schools. 

RESULTS AND DISCUSSION 

Booklet 4 

A stem-and-leaf display of the percentage correct values on the 68 
cognitive Items used in Booklet 4 is given in Table 2. The numbers in 
the first coluTin, whlcVi is lahdled "stem" list the ten digits for the 
percent correct.^ The numbers to the right provide the units digit and 
two decimal places for individual items. For example, the easiest item 
was answered correctly by 97.17% of the test takers. This is denoted in 
the first row of Table 2 by the stem of 90 and a leaf of 7.17. The fre- 
quency of 1 indicates that only one item was answered correctly by more 
eh^n^5% of the test takers (the second row lists percent correct between 
90 and^N9 5%). 

As can^x^e seen in Table 2, the item difficulties ranged in percentage 
correct from 2^.^7%. There are a large number of relatively difficult 
Items. Thirty one o^the 68 items, for example, are answered correctly 
by fewer than 40% of the^^^t takers. Appendix A. provides for each item 
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Che percent correct«, the polat-hlserlal, Sato's caution Index, and the 
laodifled caution index. Two Items were Identified with modified caution 
Indices over .5 suggesting tjiat^ an unex^^ number of low pec- 

forming students are answering these items correctly. The means, standard 
deviations, and correlations among the four-item statistics are given in « 
Appendix B. A high negativecorrelatlon was found between the polnt- 
biserial and each of the caution Indices. The extremely high correla- 
tions among the caution indices for the items (.99) is not surprising. 
The two indices yield nearly identical Information about the items. 



Insert Table 2 about here 



The intercorrelations among Sato's caution index,' modified caution 
index and total score for the total set, and two subsets of items are 
reported in Table 3. The correlations among caution Indices is .98 or 
higher for all three sets of items. The correlations of the caution 
indices with total test -score vary from being small negative (-.31) to 
small positive (.12). These correlations are consistent with other evi- 
dence that suggests that the sign and magnitude of the correlation be- 
tween total scores and caution Indices varies as a function of the 
skewness of the total test score distribution. No significant relation- 
ship was detected among the caution indices for differing sets of items. 
This was a very Interesting finding never . previously reported, which 
suggests either that the caution indices provide' unreliable Information 
or that the information is quite content Specific Results reported 
below are raost consistent with the latter interpretation. 



Insert table 3^about here 
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The scattetplot for the modified caution index computed over ^all 

items with the total score is shown in Figure 1. The ^negative relation- 

•J 

ship between the vaodified c^^ation inde.c and total score can be seen in 
the scatterplot. The presence of high caution indices for individuals 

with low total scores could possibly result from a combination of fac|:ors 

ft 

including guessing correctly on one or more of the very difficult items, 
irregular study habits for low achievers, long-term absence fr6m instruc- 
tion, wide range of item difficulties on the test, teachers stressing 
problem solving activities which improves student performance on the 
type of mathematics exercises that are considered .Jthe most difficult for' 
the nation, and teachers not covering parts of the curriculum that are 
in the NAEP" assessment of mathematic achievement. A maximum value of 
.56 was given the individual with the most unusual response pattern. 
No one with a total correct score" of 25 or , more had a caution index 
higher than .40, whereas 35 of the people with total scores less than 
25 had caution indices exceeding that value. 



Insert Figure 1 about here 

The scatterplot for the, modified caut'Yon index, computed for the 
arithaetic items with the total score is shown in Figure 2. One student 
performing very well on these items raissed a very ea^y item contributing 
to the largest modified caution index of all at .81. The scatterplot 
reveals the negative relationship between the modified caution index value 
and the total score. Similar factors as reported earlier could possibly 
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help to explain the different response patterns. 

— — — — — 

Insert Figure 2 about here 



An Interesting clian^>e in. the relationship can be seen in the .scatter- 
plot for the modified caution index computed for the georaetry items with 
the total score for geometry items (Figure 3). A positive but near zero 
r-lationship is found betwefen f. a modified caution index for geometry and 
the total geometry score. This change^ in relationship from negative to 
positive suggests the strength of the relationship between the modified 
Taution iiidex and the total test score is confounded' by "the nature of "the 
total score distribution. 

Insert Figure 3 about here 



Of particular importance and seen in the scatterplot of modified 
caution indices computed for the arithmetic and geometry items given in 
Figure A,. is the small positive near zero relationship among the modified 
caution indices across t!ie two domain of items. This result suggests 
that unusual response patterns for individuals are not consistent across 
item domains. 



Insert Figure A about here 



Since Sato's caution index was correlated .99 with the modified 
caution index, we will drop reporting of Sato's caution index in favor of 
the modiEied caution index which is bounded at 0 and ^1 . School variables 
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that were. significantly correlated in the positive direction with the 
modified caution index for the total set of items were percent of com- 
auaity on welfare, percent of community farm workers, percent of eora- . 
munlty black, and number of students qualified to receive Title I and 
are reported in Table A. School variables that were significantly cor- 
related in the negative direction with the modified caution index for. 
the total set of items were percent of community profession workers, 
percent of comiaunity skilled workers, -and type of school. 



Insert Table 4 about here 



Uo significant gender differences were foun^/with the modified caution 
index. However, the results all consistently show females as having jj'^^'' 
caution values across the different item sets. y 

The Like- Mathematics scale revealed no significant relationship with 
the caution values while. the Usefulness of Mathematics scale shov^ed a 
significant negative relationship with the caution index. Students per- 
ceiving mathematics as not very useful were students with high caution 
indices. 

Since the caution Indicfes were substantially correlated with the total 
score, partial correlations of school and student background were computed 
controlling for the respective total test score. The partial correlations 
are given In Table 5. Significant posltive'partlal correlations were found 
with the modified caution index for the total set of items on the following 
variables: percent of community farm workers, percent of community black, 
and number of students qualified to receive Title I. Significant negative 
partial correlations were found with the modified caution Index for the 



/total set of items on the following variables: percettt of community 
"^skilled or unskilled workers, and the usefulness of Mathematics Scale. 



Insert Table 5 about here 



The results of the analyses of variance and of ^ovariance (ANOVA 
and ANCOVA) are given iti Table 6. .For the total set of items and for 
the arithmetic items, the caution index varied significantly on the set 
of four variables investigated: community size, race-ethnicity, grade- 
level, and census division. The ANCOVA results, where we controlled 
for the total test score, revealed only one variable witl"; a significant 
main effect .remaining. This significant effect was fop the race- 
ethnicity variable indicating that the differences in caution indices 
were still significant between groups of students classified on this 

■ - ■ t , . , 

variable. ^ . 



\ 

Insert Table 6 about here 



Ct- 

The results from the ANOVA for the geometry set of items revealed 
that the caation inde'x varied significantly on onlf the race-ethn^city " 

* { 0 

variable. The ANCOVA results aftet conttol'l^ng for the tot^l geometry _ 
score revealed significant differences both on- the race-ethnicity and, the 
census division variable. ^ - - . • j. 

To illustrate the varying d'istrJ.butions^ of 'caution indices by total 
score for specif ic<- subgroups we preparect "Box and Wl=,isker -Plots." A box 
aU whisker plot^provldes a graphical picture of the location of the 25—, 
.SoHlL and 7S^ percentile score with thJ range of values "plus the mean . 
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value. . The' box represenLs tl\e* !^i4dle 50% of the values while the whisker 
or tails extending from the box represent the"J.ower and upper quartlle 
values. Values of 0 on tJie ibox and whisker plot indicate the probability 

^ ... ^ - 

occurrence as 1 chance out of 20 while an asterisk iindicates the^proba- 
bility of occurrencfe at 1 chance, oiit of 200, , 

• The total test score interval for the total and arithmetic set qf 
itaras were split into deciles while the tedt score interval for the geometry 
set of items was split into quintiles. Box and,.Whisker ploJi^s of the raodi- 
fled caution index. for. blacks and whites by their test . performance level 
are c^iven-in Figures 5, 6, and 7 for tb^ total arithmetic and geometry set 
of items/ respectively. Blacks across each of three total test intervals 
for each decile have higher cautiori'indlces as can be seen in Figures 5, 
6, and 7* »^ ' ... 



^ Insert Figures 5, j6, and* 7 about here 

y 

G • ,/ 

— -^-••^•««-» — — — — f 

' • ^. ' . ■ ' . ^ . ^ 

Box-^and whisker plots of the modffled cauti<^ inde^^ values across the 
test score performatfce levels for males and fema(es ^^are' given in Figures 8, 

""9, and 10 for the total, arithmetic and geometry set of items, respectively. 

' An interesting patter^n of results can be seen in each of these figures; that 
is, the females of a low ability \evel have lower fcaution values ^.t;han males 
of equal ability while females of a high ability level have greater caution 
indices^ than malea of equal ability.' The mldspread (i.e., the distance from 
the 2'>^ to the 75^ percentile) of caution ^ indices for the studenta 
in the^ first decile are -about one and a half times the midspread throughout 
the remaining deciles^ 



Insert'-Figures 8, 9, and 10 about here 
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Box and whisker plots of the modified caution index values acr;oss the 
test score* performance levels for two .groups formed based on a median split 
of the Usefii^lness of Mathematics scale are given in Figures 11, 12, 13 
for the total, arithmetic and geometry set of items, respectively. At the 
extremes of th,^ test score intervals higher cautloa values can be seen for 
students expressing little usefulness of mathematics compared to students 
who expressed a great amount of usefulness of tuatheraatics^ No apparent 
differences between groups formed *based on the usefulness of mathematics 
could be seen in the middle region of the test score interval. / 

^ Insert Figures 11, 12, and 13 about here. - - v 

Box and whisker plots of modified caution index values were a\so 
obtained and inspected for a high and a low group of students based on 
a median split of the Like Mathematics scale. No major differences were 
found that would necessitate their figures to be included. 

Since the differences >^n modified caution index values noted on the 
box and 'Whisker plots for blacks and whites could be Influenced by total 
test score, we evaluated the residuals remaining after regressing the 
-modified caution' index on total test score. Box and whisker, plots of the 
residuals across the test score intervals^, are given for blacks and whites 
in Figures 14, 15, and 16 for the total, arithmetic and geometry set of 
items. The pattern of blacks higher residuals. can be . seen in the figures 
across all three test score Intervals. This pattern of .differences in 
response patterns between blacks and whites after controlling for test 
score Indicate that groiips of students differ in their response patterns 
even after adjusting for differences in total correct scores. On average. 
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the response patterns for black students are less consistent with the 
national norm than are the response patterns of white students with com- 
parable total number correct scores. 

Insert Figures 14, 15, and 16 about here 

Sch ool and Census Division Differences on Achievement and on the Modified 
Caution Index 

To determine the variation in achievement and modified caution Indices 
that can be explained by census division and school differences within 
census divisions we divided* the variation among the pupil's achievement 
test scores and modified caution index values into a betweafi- census divi- 
sion and a within-^census division component. Table 7 shows the percent 
of variation explained for pupil's mathematics achievement and modified 
caution index valfu6s on the total, arithmetic and geometry set of items. 



Insert Table 7 about here 



We will now focus on the results of the variance decomposition of 
the mathematics achievement total correct scores and the modified caution 
index for the total, arithmetic and geometry set of items. Between census 
division variance on achievement ranged fiom 2.9% of total variance on the 
arithmetic items to. 3.5% on arithmetic items. Schools within-census divi- 
sion achievement variance ranged from a high of 25.5% on geometry items to 
a low of 22-5% on arithmetic items. For the total set of items 4% of the 
total variance was explained by between census division variance and 28.2% 
explaliierl by scliools within census division. Clearly , there are sizeable 

29 ■ 
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between school differences In achievement, a result that could have been 
predicted with great confidence without the analysis. But It Is also 
clear that the within school variability In achievement accounts for a 
much larger fraction of the total variation. 

Between census division variance on modified caution Indices ranged 
from .05% of the total variance on arithmetic items to .43% on geometry 
items. Schools within census division modified caution index variance 
ranged from a high of 7.8% on arithmetic items to a low of 1.9% on 
geometry items. Schools within census division explained 11.8% of the 
total variance while only .8% was explained by between census divisions. 
The between school variability in the caution indices is noticeably less 
than that for total scores. Nonetheless, there are sizeable between 

school differences in caution indices for the total set of items and for 

■ ■■■ I 

the set of arithmetic items. These between school differences suggest 
significant ivariation between schools in content emphasis. ^ 

The modified caution indices for all three tests were used as de- 
pendeilt variables in a hierarchical ANOVA. The first factor is the 
nine different census divisions of the nation. School is the second 
factor. This factor is nested in census divisions, while students, the 
third factor, are nested within schools within census divisions. The 
results of these analyses are summarized in Table 8* The schools within 
census divisions 3, 6, 7, and 9 have significantly different modified 
caution indices on the total item set. ^ _ ^ 



' Insert Table 8 about here 
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The mean caution indices on all tests for the 13 schools in census divi- 
sion 6 and 15 schools in census division 7 are reported in Tables 9 and 10 
respectively. The range of the school mean modified caution indices is .15 
to .27 on all items. Tha relatively wide range of modified caution indices 
for schools within these census divisions suggests a high^degree of varia- 
bility of item response patterns. These large differences as noted in 
Tables 9 and 10 may well be a function of the curriculum. The school effects 
are significant for all indices, which reveals that certain schools may not 
have covered segments of the content sample on the test, or that they may 
have given less than typical emphasis to some of the content. 

Insert^Tables 9 and 10 about here 



The significant schools-within-census division effects denote the 
high degree of variability of student performance in schools within 
certain census divisions of the country. Curriculum offerings may very^ 
well contribute to these large differences. To explore this possibility, 
we conducted a more detailed analysis of response patterns of students 
at schools within census division 6 and 7 following the approach used 
by Harnisch and Linn (1981). This study was designed to identify the 
subset of items v/hich contribute most to the caution indices for the _ 
use to describe unique patterns of performance by item content. Various 
patterns of performance suggest differences in content coverage that 
make, the test less appropriate for some schools than others. 

Schools from censu? division 6 and 7 with 10 pr more students Were 
evaluated for unusual response patterns. The p^values, i.e., the pro^ 
portion of students who answered an item correctly, were computed for 
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each school on each of Che JfS Items- Since school mean performance on 
the test Is directly related to the p-values on the items, a linear re- 
gression was performed on the p-values for each school with the p-vllues 
from the nation. The regression equation was used to compute the ex- 
pected proportion correct on each item for each school. Residual scoies 
were computed simply by subtraction of expected from observed proportion 
correct on each Item for each school. 

Items were categorized In terms of their content and format In orde 
to find clues about the possible reasons for the large differences In 
the residuals. The mean of the residuals for each category was then 
standardized by dividing by the standard error of estimate. Finally, 
the standardized mean resldualsT were multiplied by the square root of 
the number of items in the content category as a means of weighting the 
standardized mean residuals according to the number of items in the 
category. Tlie resulting lighted standardized mean residuals, which are 
analogous to critical ratios, were used to compare the items in dif- 
ferent categories. These results are reported in Tables 11 and 12 for 
the two census divisions. 



Insert Tables 11 and 12 about here 



An entry in Table 11 or 12 greater than 2.0 in absolute value indi- 
cates that items in that particular category are much easier or much 
harder for students in that school than would be expected from their 
overall performance and the relative difficulty of .these items for tfhe 
national sample as a whole. Five of the content catf^gorles for schobls 
within census division 6 have entries in Table 11 greater than the 2.0 
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absolute value for one or more schools. Seven of the content categories, 
for school within census division 7 have entries in Table 12 greater 
than the 2.0 absolute value for one or more schools. The one category 
of items in which two or more schools from either census division 6 or 7 
had large, entries was sign numbers. For'^schools .within census division 
6, the large positive entry (3.88) for School 2 for the sign numbers 
stands in marked contrast to the large negative values for Schools 3 and 
4 in this category (-2.45 and -2.23 respectively). This suggests the 
hypothesis that the use of sign numbers may be quite common in School 2 
but rare in Schools 3 and .4. Similar hypotheses are suggested by the 
other large values in Table 11 and Table 12. 
Booklet 5 

A stem-and-leaf display of the percentage correct values on the 
63 cognitive items used in Booklet 5 is given in Table 13. The item dif- 
ficulties ranged in percentage correct from 2 to 95%. Twenty of the items 
are answered correctly by greater than 70% of the test takers while nine- 
teen of the items are answered correctly by fewer than 40% of the test 
takers. Appendix C provides for each item the percent correct, the point- 
biserial, Sato's caution index, and the modified caution index. Two 
items were identified with modified caution indices over .5 suggesting 
that an unexpected large number of low performing students are answering 
items correctly. Tlie means, standard deviations, and correlations among 
the four-item statistics are given in Appendix D. The caution indices 
» correlated as low or lower with the percentage correct as compared with 
' the; correlation' of the point^blserial with the percentage^ correct. The 

correlation of the caution indices with the point-biserial was a high 
, ^-negative value. The extremely high correlations among the caution indices 
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for the items (.00) is nor surprising as was noted earlier with reference 
to Book 4 items. It can be stated that the two indices yield nearly 
identical information about the items. 

Insert Table 13 about here 



The intercorrelations among Sato's caution index, modified caution 
index and total score for the total set and two subsets of items are 
given la Table 14. The correlation among caution indices is .99 for all 
three sets of items. The correlations of the caution Indices with total 
test score varies from being small negative (-.21) to small positive (.01). 
The correlations between the caution indices and total test score for 
arithmetic and geometry items are nearly zero. Since the relationship of 
caution indices with test score overall items is negative, this suggests 
that the non-arithmetic and non-geometry items are contributing substan- 
tially to this correlation. The relationship of caution indices across 

1 

different item subsets was nearly zv^ro. This concurs with our finding 
from Book 4 and suggests that the information is quite content specific. 
The results reported for Book 4 and the below are most consistent with 
this interpretation. 

Insert Table' 14 about here 
***— — ^ f , 

The Rcatterplot for the modified caution^ Index computed over all 
items with the total test score is shown in Figure 17. The ,8mall negative 
relationship of modified caution values with the total test score can be 
seen in the scattecplot. As noted earlier, and worth repeating, is the 
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notion that high caution" indices 'for individuals can result from a com- 
bination of many different factors. These large caution indices possibly 
result from a student guessing correctly on one or more of the very dif- 
.Hcult items, irregular study habits for low achievers, long-term absence 
from instruction, wide range of Item difficulties on the test, teachers 
stressing problem solving activities which improve student performance - 
on the type of mathematics exercises that are considered the most dif- 
ficult foc the nation, and teachers not covering parts of the curric»^lum 
that are in the NAEP assessment of mathematics -achievement . A maximum 
value of .58 was given the individual with the most unusual response 
pattern. No one with a total correct score of 28 or more had a caution 
index higher Chan .40, whereas 6 of the people with total scores less 
than 28 had caution indices exceeding that value. 



Insert Figure 17 about here 



The scatterplot foi" the modified catuion index computed for the 
arithmetic items with the total score is shown in Figure 18. One student 
answering only two items correctly and perhaps nearly the most difficult 
arithmetic items had the largest modified caution index value of .88. 
The sdatterplot reveals a broad range of modified caution values across 
the test score interval with only a few unusually high values spread 
across the test score distribution. A host of factors as noted earliei: , 
could possibly help to explain the difeei;ent response patterns. 



Insert Figure 18 about here 
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The scatterplot for the modified caution index computed for the 
geometry items with the total score fo.r geometry items is given in 
Figure 19. The scatterplot reveals the near zero relationship between 
the modified caution index and the total geometry test score. Modified 
caution index values exceeding .90 are found for five students scoring 
nine or above on. the geometry items. The near zero relationship as seen 
from the scatterplot indicates that the caution index values a-te pro- 
vlding us with new information not contained, in the total test score* 
An interesting area of research is trying to understand the factors 

associated' with students that have large caution values or schools with 

t 

large. numbers of students who have large caution values* 



Insert Figure 19 about here 



No systematic relationship was found between the modified caution 
index computed on arithmetic ftems and the modified caution index com- 
puted on geometry items. The scatterplot that reveals this near zero 
relationship is given in Figure 20. An individual with a large caution 
value on one subset of items is not necessarily going to have a large 
caution value on a different\et of items. 



; Insert Figure 20 about here 



Since the relationship between Sato's cautio^n index and the modified * 
caution index i/as nearly one, we will continue, reporting only, the modified 



caution index sjlace it is bounded 0 and 1. The correlattoas between a 
set of school, student backgrouad and affective dotnala variables and the 
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modified caution index and total test score are given in Table 15 for the 
total, arithmetic and geometry sets of items. S^x is found to be the 
variable most associated with the caution Indices. Feraal.es displayed 
consistently lower caution values across the different Item sets. The 
two school variables found to be associated with modified caution Index 
values are percent of coiomunity on welfare and percent of community black, 
each in tx positive manner. 



Insert Table 15 about here 



For all practical purposes, no significant relationship was found 
between the affective domain scales and the modified caution Index. How- 
ever, the validity coefficients for all the affective domain variables 
with the respective total test score are all positive and range from 
.06 to .31. The easiness of mathematics scale consistently correlates 
.20 or higher with each^of the total test scores. 

The results of the ANOVA and ANCOVA are reported in Table 16. Sig- 
nificant differences in caution indices for the total set of items were 
found for groups formed based on their race-ethnicity and grade level 
variables. No significant differences were found between groups for the 
modified caution index computed on the arithmetic items. On the geometry 
items, the three variables found to indicate large between-group dif- 
ferences on the modified caution index were community size, race-ethnicity 
and census division. The ANCOVA results on the geometry items, where we 
controlled/for the total geometry test score, revealed significant, between- 
group differences on each of the three variables noted above. On the 
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other hand, the ANCOVA results on the total set of items revealed no sig- 
nificant between group dif f e).\^hces after controlling for the total test 
score* 



Insert Table 16 about here 



The distributions of caution indices by total test score for specific 
subgroups were obtained and inspected. Box and whisker plots were pre- 
pared to illustrate^the conditional distributions of caution indices for 
the total test score interval split into deciles for the arithmetic and 
total set of items while the geometry test -score interval was split into 
quintiles. Only two of these plots will be included. * 

Box and whisker plots of the modified caution index values across 
the test score performance levels for males and females are given in 
Figures 21 and 22 for total and arithmetic set of items, respectively. 
The pattern of results shown in Figure 21 is consistent for the males 
across the test performance levels with males having gjeater caution 
indices- For the arithmetic s'et of itams> a similar pattern of results 
are found at the extreme performance levels while some moderation of 
differences in caution indices occurs at the middle range of the per- 
formance levels. 



Insert Figures 21 and 22 about here 



School and Census Division Differences on Achievement and on the Modified ' 
Caution Index , - 

To determine the variation in achiev.ement and modified caution Indices 
that can be explaLaed by ceusus division and school differences within ^ 

:■ - 38 . ' .• 
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census divisions we "arvided the variation among the pupil's achievement 
test scores "and modified caution index values into a between-census divi- 
sion and a withln-census^ division component.. Table 17 shows the percent 
of variation explained for pupil's mathematics achievement and modified 
caution index values on the total, arithmetic and geometry set of items. 



Insert Table 17 about here 



The variance decampositlon of the mathematics achievement total 
correct score fo^' the total, arithmetic and geometry set of items into ■ 
a betwee.n census divfsions component and a schools within census divisions 
revealed similar percentage patterns as were found with Book 4. Be.tween 
census division variance on achievement ranged from 2.0% on the geometry 
items to 4.3% on the arithmetic items. Over two times as much of the 
variance In achievement was explained for the arithmetic items compared 

* f 

to the geometry items with the schools within census divisions component 
of variation. For the total set of items 5.2Z of the total variance was 
explained by between census division variance and 23,3% eScplained by 
schools within' census division. In each of the subsets and also for the 
total set of items the percentage of variance occurring within schools 
is much greater than between schools, revealing a wide range of -indi- 
vidual differences within schools. 

The schools within census division modified caution index variance 
was nearly the , same on the arithmetic and geometry set of items, 2i77. 
and 2.3Z, respectively. For the total set of items, schools within 
-census division explained 3. 5%. of the .total variance. Differences at 
• thtv between school level on the caution indices suggest significant 
variation between schools in emphasis given to the subject mat.tec. . , 
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The modified caution indices for all three tests were used as d.e- 
pendent variables in a hierarchical ANOVA. Tfie first' factor is the nine 
different census divisions o^ the nation. School" is the second factor. 
This factoY is nested^in census divisions, while ^tudents, the t^iird 
factor, are bested within schools within census divisions. The results 
of these analyses are summarized in Table 18. The schools within census 
divisions 2 and 9 have significantly diffierent modified caution indices- 
on the total item set. For the geometry item set,, schools within census 
divisions 1 and 3 have significantly different modified caution indices. ' 



Insert' Table 18 about here 



. ^ Insert Tables 19 and 20 about here 



\ 



The mean caution indices on all tests for the 8 schools in census 
division 1 and 21 schools in censua division 9 are reported in Table 19. 
and 20^ respectively. The range of the school mean modified caution 
indices is .03 to ^25^ on the geometry set of items foj: schools in census 
division 1 and .11 to .28 on the arithmetic set of litems for schools 

with'in census division 9'* The* relatively wide range of modified caution ^ 

indices for schools within these census divisions suggests a high degree 

s 

of variability of item response patterns. These large differences as 
noted in Tables 19 and 20 may well have been a function of the curricu- 
lum. The school' effects are significant for all Irfdices, which reveals 
that certain schools may not have covered segments of the content sampled 
on the test, or tWt they, have given less than. typical emphasis to some 
of the content. . ^ . 



The stgnif leant schools-wi Chin-census division effects denote the 
high degree of variability of student performance, in schools within 
^r^aln^census"dlvlsions"of the country. " Cuf riculum of f e"r inp nay-very 
well contribute to these large differences. To explore this possibility, 
we conducted a more detailed analysis of response patterns of students 
at schools within census division 1 and 9 following the approach used by 
Harnlsch and Linn (1981). This analysis was designed to identify subsets 
of Items which contribute most to the caution indices for the respective 
schools. The analyses of these subsets of items were used to describe 
unique patterns of performance by item content. Various patterns of per^ - 
forraance suggest differenr.es in content coverage that make the test less 
appropriate for some schools than others- 

Schools from census divisions 1 and 9 with 10 or more students were 

1.0 

evaluated for unusual response patterns. The p-values, i.e., the propor- 
tion of students who answered an item correctly, were computed for each 
school on each of the 68 items. Since school mean performance on the. 
test is directly related to the p-values on the items, a linear regres- 
sion was performed on the p-values for each school with the p-values from 
the nation. The regression equation was used to compute the expected 
prc^portlon correct on each item for each school. Residual scores were 
computed simply by subtraction of expected from observed proportion 
correct on ekch* item for each school. 

Items were categorized in terms of their content and format in. order 
to find clues about the possible reasons for the large differences Jn the 
residuals. The means of the residuals for each category were then 
standardized by^Tivttfing by the standard error of estimate. Finally, 
the standard-i-zed meaiv residuals , were multiplied by the square coot of 



the number of Items In the content category as a means of weighting the 
standardized mean residuals according to the number of items in the cate- 

gory» The res^u-tti-n^-weighted standardized mean— residuals^ which-are 

analogous to critical ratios, were used to compare the Items in different 
categories. These results are reported in Tables 21 and 22 for census 
divisions 1 and 9, respectively. 

I 

Insert Tables 21 and 22 about here 



An entry in Table 21 or 22 greater than 2.0 in absolute value indi- 
cates that items in that particular category are much easier or much 
harder for students in that school than would be expfe'tted from their over- 
all performance and the relative difficulty of these items for the national 
sample as a whole. Five of the content categories for schools within 
census division 1 have entries in Table 21 greater than the 2.0 absolute 
value for one or more schools. Story problems is the only content cate- 
gory that does not have entries greater than 2.0 absolute value for one 
or more schools within census division 9. The f ractions-to-decimals cate- 
gory of items had three or more schools from either census division 1 or 
9 with large entries. For schools within census division 1, the large 
positive entry (2.81) for School 1 for -f racttons^to-decl^mals stands in 
marked contrast to the large nega^tive value for Schools 5 and 8 in this 
category (-3.09 and -3*.35, respectively). This suggests that the hypothe- 
sis that the practice of transforming fractions to decimals nnay be quite 
common in School 1 but rare in Schools 5 and 8. Similar hypotheses fare 
suggested by the other large values in Tables 21 and 22. Thus, one would 
expect that special emphasis is placed on performing calculations in; 
Schools 14, 16 and. 19 from census division 9. 



SUMMARY AND IMPLICATIONS 

Sa^to^-s~caut~ion"i"nd«x-^nd^ 

puted for the students and the test items from two test booklets from the 
1978 math assessment for the 13 year old sample. The correlation between 
the modified caution index and Sato's caution index was nearly one both 
for the indices computed for the students and the problems. The remainder 
of the analyses focused on reporting the variables associated with the 
modified caution index. Comparisons of subgroup caution index values 
were used to identify student background characteristics that were asso- 
ciated with unusual response patterns. 

Near zero correlations were found between the caution indices com- 
puted for different subsets of Items. Further analyses revealed that 
the information contained in the modified caution index was quite content 
specific rather than providing unreliable information about an individual. 

Significant gender differences were found on the caution Indices 
revealing that females tend to have smaller caution values across, the 
total test score interval. Significant race-ethnicity differences were 
found on the caution indices with blacks having larger caution values 
across the total test score interval. 

Significant differences in caution indices, namely lower values, 
were found associated with students having a perception that mathematics 
was not very useful. This relationship was quite apparent throughout 
the total tast score Interval. 

Differences in response patterns at the school level were identified 
and specific subsets of items were identified as contributing to these 
large caution values. The association of subsets of items with large 
caution indices at particular schools has potential diagnostic value- 
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While numerous reasons may be given to explain the school differences 
in caution indices it is quite plausible that the differences in response 
patterns result from variability in content coverage and emphasis. This 
possibility deserves further ^tudy. 

From a research point of ylew, it would be most Interesting and in- 
formative to extend the analysis to the National Assessment of Educational 
Progress sample^ of 9- and 17-year-olds to make further comparisons of sub- 
groups on the caution index values as well as identifying student back- 
gtound characteristics that are associated with students having unusual 
response patterns* - 

Differences in response patterns found in this study* indicate that 
there are Individuals for whom the total test score conceals systematic 
deviations from the typical pattern of responses and may be misleading. 
The results also suggest that there are schools with curricula that do 
not match the test content. Formation of a contirigency table as sug- 
gested by Sato (1975) for the caution index along with the total test 
score can be used to identify students who need more study, who make 
careless mistakes, who possess sporadic study habits or insufficient 
readiness or who are doing everything fine. Schools can similarly be 
identified as having in general one or more of the above categories of 
students. 

/ 
/ 

I ■ 
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Notes 

!• Tatsuoka, K., & Tatsuoka, M. M. Detection of aberrant response patterns 
and their effects on dimensionality > (Resear ch Report 80-4) Urbana, IL: 
~~ 'U^tv^rsrty"oFTITIt^^ 
1980. 

2. Public Use Data Tapes can be purchased from; 

Department of User Services 

National Assessment of Educational Progress 

Education Commission of the States 

1860 Lincoln Street, Suite 700 

Denver, Colorado 80295 

3. Tatsuoka, M. M. Recent psychometric developments In Japan: Engineers 
grapple with educational measurement problems . Paper presented at the 
lONR Contractors Meeting on Individualized Measurement, Columbia, MO, 

^ 1978. 
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Exhibit 1 
Book A Affectiv e Doma i n Items 



Like Scale Items 

1. Feel like taking mathematics tests 

2. Feel like doing mathematics homework 

3 . Fe e 1 1 lke_Jiel4> lng--a"-c-l:a:sBmaTe^ tics 

4. Feel like playing mathematics games 

5* ,Feel like listening to teacher explain math 

6* Feel like watching teacher work at the board 

?• Feel like using hand calculators in math 

8, Feel like using a computer in math 

Usefulness Scale Items 

1« Feel useful to take raathematXGS tests 

" ' 2« Feel useful doing mathematics homework 

3. Feel useful halping a classmate do math 

4. Feel useful to play mathematics games 

5. Feel useful to listen to teacher explain math 

6. Feel useful to watch teacher work at the board 
?• Feel useful to use hand calculators in math 

8» Feel useful to use a computer in math 



• > 


■ ' ■ ■•: - . ■ . M : \ 






B 


Exhibit 2 \^ 




Book 5 Affective Domain Items ' ^ 


St;udents 


rated each of the mathematlcs^a^^ivltles -below with reference 


— ■ ' ' , ■ ■ 


^to like 


for. Importance, and easiness. 


1. 


Solving word problems 


2. 


Working with fractions 


3. 


Estimating answers to problems 


A. 


Measuring lengths, weights or volume 


5. 


Working with metric measures — 


6. 


Doing proofs — ^ 
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Table 1 

S-P Table for 18 Examinees and 5 Items 
<Hypothetlcal Examplife) 



Exaralaee 


» 




- -2^- 


Item 
j 

1 4 


5 - 


Examinee 
Total ■ 


Sato's 
< CButlon 
Index 


Modified 

L»clul-XOIt 

Ir^dex 




















— ' — r 


• a 

1 


1 




1 


0 


^ -A 


• UU 


• UU 


2 


/ 


r 


1 


1 

L 


0 


1 , 




• 03 ' 




3 

' 4 .- 




1 
1 


1 
1 


■ 1 

0 


0 

:1 


0 

1 0 


J 


• UU 

1 /I 

• io 


• UU 

• Uo 


5 


• • > 


1 


1 


0 


o' 


i ^ 


3 


• O J 




V 

6^ - 




1 


0 


1 


0 


I 1 


J 


1 1 ^ 


• J** 


1 




1 


1 


0 ^ 


r P 


• 0 


o 
Z 


no 


on 




" J , 

/ 


^ 1 * 


1 

1 1 

1 » 
1 


0 


0 


0 


/ • 2 


no 


.00 


•9 




1 


0.. 


1 




0 


9 




.23 






1 


1 

. .0. . 


0 


1 


0 








' " ' 11 




0 I 


1 


1 


0 S 0 


2 


.74 


.39 


12 




0 I 


1 


0 


1 


0 


2 


.88 


.47 


13 




1 


0 


0 


0 


0 




.00 


.00 


14 




- 1 


0 


0 


0 


0 




.00 


' .00 


15 




0 


1 


0 - 


- 0 


0 




.45 


.22 


16 




0 


0 


1 


0 


0 




1-.14 


.56 


17 




0 


0 


0 


1 


0 




1.36 


.67 


' 18 




0 


0 


0 


1 


0 




li36 

L 


.67 

9 



Item Total ^ n . 12 10 7 6 3 

. •J 
Sato's Caution 

Index .M .28 .42 .95 -21 

Modified Caution ^ 

Index ' C. .14 -14 ,21 -50.. 13 

^ J 



Table 2 

Stem and Leaf Display of p Values for Cognitive Items- 



Book 4, 13-year olds, 1978~ 
(N - 68) 





Stem 


Leaf 












ft rv 

90 


7.17 












yO 
Q r\ 

oO 


4.54 ^ 1.10 
8.14/ 6.83 




5.02 




• 


80 


1 "4Ti6 


4 • Oo 




^ . ox 






70 


9.85 


9.48 


Q LL 








70 


1.69 


1.48 










60 


8.24 


5.90 


^ 7R 
J . / o 








OU 


4.87 


3.97 










50 


ihi 


7.00 


5.31 








jU 


4.78 


4.37 


3.10 


0.96 


0.55 


0.14 


40 


9.65 


7.97 


6.37 


C OA 

5.84 




» 


40 


4.36- 


3.33 










30 


9^68 


8.65 


5.70 


5 1.3 






30 • 


4.39 


3.28 


3.03 


2.87 


2.50 


1.68 


20 


9.13 


8.48 


8.44 


6.14 


5.77 


5.24 




.4.33 


4.29 


3.43 


3.14 


'2.65 




I 


9.45 


9.45 


7.28 


6.78, 


6.50 




10 \ . 
0 \ 


7.35 












0 \ 


3.90 


2.05 











Frequency 



1.23 0.57 



1 

2 

4 

4 

3 

2 

4 

2 

3 

6' 

4 

2 

4 
8 
6 
5 
5 
0 
1 
2 

68 



\ 0 
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" \ Tabl-e 3 < • 

Means, Standard. Deviations, Skewness, and the Pearson Product Moment 
C orre l a t lo n s Among Sat o's C aut ion Index, Modified Caution Inde x and the 



Total Test Score for the Total Set and Two ^Subsets of Items 
Book A,- 13 year olds, 1978 
. (N = 1219) . ' 

\ . ■■ 

Variable • 



V Variable 


1 


2 


3 


A 


i 5 


6 


7 


8 


9 


All T Am A 

All items 






- 














1. Sato* 8 Caution Index 


1 nn 
X • uu 




- 1 9 


.74 


.74 


-.IS 


.51 


.51 


-.01 


2. Modified Caution Index 




1.00 


-.31 


.74 


.76 


-.27 


.48 


.49 


-.11 


3. Total Test Score ^ 

•» .} 






1.00 


-.17 


-.31 


.95 


.05 


-.02 


.ol 


Arithmetic Items' 




/ 

/ 










• 






A. Sato's Caution Index 








1.00 


.98 


-.17 


.07 


.08 


-.oa 


5. Modified Caution Indes 










1.00 


-.32 


.07 


.OiB 


-.18 


6. Total Test Score 












1.00 


.03* 


-.02 


.64 


Geometry Items 












* 








7. Sato's Caution Ind^ex 














1.00 


.99 


.12 


8. Modified Caut5^on/Index 
















1.00 


.04 


- 9. Total Test Score 


















'1.00 


Mean 


.35 


.18 


33 ".00 


.29 


.15 


16.79 


.40 


.20 


8.30 


• Standard Deviation 


.13 


.07- 


ia.56 


.16 


.09 


6.20 


.23 


.12 


3.28 


Skewness 


.91 


-1.10 


.28 


1^6 


1,.26 


.31' 


.70 


.74 


.58 










•(J 













Table A 



Pearson Product Monient Correlations* of School, Student Background, 
and Affective Domain Variab^ with Modified Caution Index and the 
Total Test Score for the Total Set and Two Subsets of Items 
Book a; 13 year olds, 1978 
(N = 1219) 







All Items 


Arithmetic 


Items 


Geometry 


Items 






Modified 


Total 


L lUU X X X C U 


X U Ucl X 


Modified 


Total 


Variable 


Caution 


Score ■ 


Caution 


Score 


Caut'^ion 


Score 


1 ■ 


reiTcenu or v>oruiiiuriiuy 
unskilled 


-.04 


-.05 


-.02 


-.05 


-.02 


-.07 


2. 


Percent of community / 
not employed 


.07 


-.14 • 


.05 


-•12 


• U L 


— TA 

— • in 


3. 


Percent of community 
on welfare 


.17 


-.32 


.16 


-•29 


.03 


-.27 


4. 


Percent of community 


.10 


— . iU 


.06 


-.10 


!o4 


-.03 


5. 


Percent of community black 


.19 


-.28 


.17 


-.26 


.05 


-.21 


6. 


^Percent of community 
professional workers 


-.10 


.29 


-.10 


.25 


.01 


.27 


7. 


Percent of community 
skilled workers 


-.12 


.18 


-.09 


.19 


-.06 


.11 


8. 


Number of students quali- 
,fied to receive Title I 


.13 


-.24 


.12 


-.23 


-.01 


-.16 


9. 


School enrollment size 


.01 


-.01 


.01 


-.02 


-.02 


.01 


10. 


rS ^ 

Sex 


-.05 


-.06 


-.05 


-.01 


-.02 


-.09 


11. 


Family owns calculator 


-.09 


.19 


-.05 


.17 


-.02 


.13 


12. 


Type of school^ 


-.09 


.14 


-.05 


.16 


-.06 


.06 


13. 


Like mathematics scale 


.03 


-.04 


.03 


-.03 


.03 


-.01 


14. 


Usefulness of mathematics 
scale 


-.14 


.24 


-.08 


.24 


-.06 


.15 



^Sex is coded 1 for males and 2 for females- 

^Family owns calculator is coded 0 for no aad 1 for yes. 

^Type of school is coded 1 for public and 2 for private Catholic. 

^Correlations greater in absolute magnitude than .07 are significant at p < .01. 
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Table 5 

Partial Correlations* of School, Student Background, and 

Affective Domain Variables with Modified Caution Index Controlling 

for the Total Test Score for the Total Set and Two Subsets of Items 

Book A, 13 year olds, 1978 
(N = 1219) 



Variable 



All Items Arithmetic Items Geometry Items 



1. Percent of Community 
unskilled 

2. Percent of community 
not employed 

3. Percent. of community 
on welfare 

4^Percent of community 
farm workers 

5. Percent of community black 

6. Percent of community 

professional workers 

Percent of community 
skilled! workers 

Number of students quali- 
fied to "receive Title 1 

School enrollment size 
a 

Sex I 

Family owns calculator 

, c 



7. 

8. 

9. 
10, 
11. 
12. 
13. 
14. 



Type of school 

\ ' 

Like matheraaitics scale 



Usefulness 
scale 



jif mat;'hematics 



-.07 

.03 

.06 

.09 
.11 

-.01 

-.07 

.07 
.01 
-.04 
-.04 
-.^06 
.02 

-.08 



-.05 

.01 

.06 

.05 
.08 

-.02 

-.03 

.05 
.00 
-.04 
.00 
.00 
.01 

-.01 



-.02 

.02 

.04 

.05 
.06 

-.01 

-.07 

.00 
-.02 
-.01 
-.03 
-.07 

.03 

-.07 



Sex is coded 1 \for males and-2 for females. 



^Family owns calculator is coded 0 for no and 1 for ,yes. 
^Type of school iW coded 1 for public and 2 for private Catholic. 
♦Correlations greater in absolute magnitude than .07 are significant at p < .01 \ 
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Table 6 

Ahalysls of Variance Summary Results for the Modified Caution Index 

wltl? the Test Score as a Covariate for the Total Set and Two Subsets of Items 

Book 4, 13 yea- olds, 1978 
(N = 1219) 



Source 



df 



AtJOVA 

Main Effect 

Community size 3 
Residual 1214 



All Items 
Mean 
Square 



Arithmetic Items 
Mean 
Square F 



,002 
.005 



4.58** 



.021 
.007 



2.84* 



Geometry Items 
Mean 

Square F_ 



.034 
.014 



2.52 



ANCOVA 

Covariate 
Main Effect 

Community Size 
Residual 



3 

1213 



.573 

.010 
.004 



129.95** 
2.18 



.906 

.004 
.007 



135.72** 
.58 



.031 

.033 
.013 



2.29 
2.44 



ANOVA 

Main Effect 

Race-Ethnicity 
Residual 



3 

1214 



.123 
.005 



26.73** 



.124 

.007 



17.37** 



.056 
.014 



4.18** 



ANCOVA 

Covariate 
Main Effect 

Race-Ethnicity 
Residual 



3 

1213 



.573 

.047 
.017 



132.76** 
10.91** 



.906 

.042 
.007 



137.68** 
6.43** 



.031 

.074 
.013 



2.31 
5.50** 



ANOVA 

Main Effect 

Grade Level 
Residual 



4 

1213 



.046 
.005 



9.58** 



.028 
.007 



3.79** 



.016 
.014 



1.15 



ANCOVA 

Covariate 
Main Effect 

Grade Level 
Residual 



4 

1212 



.573 

.008 
.004 



129.90 
1.74 



.906, 
.007 



135.88** 
X 1.06 



.031 

.023 
.014 



2.29 
1.68 



ANOVA 

Main Effect 

Census Division 8 .015 
Residual 1209 .005 



3.19** 



.014 
.007 



1.86 



.023 
.014 



1.69 



ANCOVA „ ^ 

Covariate 1 -573 130.23** .906 136.31** .031 2.30 

Main Effect 

Census Division 8 .008 1.76 .010 1.50 .027 1.96** 

Residual 1208 .004 .007 .013 . 



ERIC *p< 5 ; 

ifflBsfiaa **p < .01 ^ w 

. . . . i 



Table 7 

Percentages of Variation for the Components in 
Mathematics Achievement and on the Modified Caution Index 
Book 4, 15 year olds, 1978 





Between Census 
Division 


Schools Within- 
Census Division 


Error 


Mathematics Achievement 








' All items 


3.97 


28.22 


67.81 


Arithmetic items 


2.92 


25.51 


71.57 


Geometry items 


3.46 


22.51 


74.57 


Modified Caution Index 








All items 


.76 


11.77 


87.47 


Arithmetic items 


.03 


7.83 


92.14 


Geometry items 


.43 


1.89 


97.68 



\ 



Table 8 

Sunimary of F Ratios for Hierarchical Analyses of Variance for the 
Modified Caution Index on the Total Set and Two Subsets of Items 





% 




Book 4, 


13 


year olds, 


1978 


i 


Effect 








All 
Items 


Arithmetic 
Items 


Georad-try 
Ite|ms 


Census Division 








5.58* 


2.65* 


1 

2.47 

1 


Schools within Census Division 




3.13* 


2.22* 


1.36* 

1 


Schools 


within 


Census 


Division 


I 


2.27 


3.06* 


1.3l\ 


Schools 


within 


Census 


Division 


2 


1.57 


1.43 


' 1.17^ 


Schools 


within 


Census 


Division 


3 


3.69** 


2.88** 


1.07 


Schools 


within 


Census 


Division 


4 


1.13 


1.31 


1.36 


Schools 


within 


Census 


Division 


5 


1.58 


1-19 


.82 


Schools 


within 


Census 


Division 


6 


4.42** 


3.41** 


,1.83 


Schools 


within 


Census 


Division 


7 


5.40** 


4.42** 


1.70 


Schools 


within 


Census 


Division 


8 


1.36 


.68 


.88 


Schools 


within 


Census 


Division 


9 


5.11** 


1.66 


2.03* 



\ 

*p < .01 \ 
**p < .001 



ERIC 
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Table 9 





Means and Standard 


Deviations 


on Modified 


Caution 


Indices 






on 


the Total 


Set and 


Two 


Subsets of 


Items f,or 










Schools 


Within 


Census Division 


6 










Book 


4, 13 year 


olds, 1978 








School 


Sanple 


AIT 


Items 




Arithmetic 


Items 


Geometry 


Items 


Number 


Size 


Mean 


S.D. 




Mo a n 


S.D. 


Mean 


S.D. 


1 


17 


.17 


.06 




.15 


.09 


.18 


.10 


2 


1j 


1 Q 


.05 




.18 


.07 


.21 


.10 


3 


15 


.17 


.08 




.17 


.10 




.09 


4 


13 


.19 


.07 




. .17 


.10 


.17 


.13/' 


5 


14 


.18 


.08 




.14 


.11 


.24 


/ 

/ 
/ 

/ 

il5 

/ 
1 


6 


14 


.23 


.07 




.19 


.09 


.30 


7 


14 


1 7 


.06 




•15 


.08 


.18 


j.io 


8 


13 


.22 


.09 




.20 


.14 


./o 




9 


18 


.15 


.05 




.12 


.07 


.19 


.11 


10 


16 


.16 


.07 




.12 


.06 


.23 


.17 


11 


19 


.18 


.06 




.15 


.09 


.18 


.10 


12 


15 


.27 


.13 




.26 


.14 


.26 


.12 


• 13 


13 


.25 


.13 




.22 


.13 


.23 


.12 



ERIC 
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Table 10 

Means and Standard Deviations on Modified Caution Indices 
on the Total Set and Two Subsets of Items for 
Schools Within Census Division 7 - " 
Book A, 13 year olds, 1978 



School Sample 
Number Size 



All Items 
Mean S.D. 



Arithmetic Items 



1 

2 
3 
A 
5 



16 
16 
15 
15 
14 



.20 
.16 
.19 
.19 
.16 



.09 
.07 
'.06 
.09 
.08 



Mean 

.16 

.11 

.16 

.14 

.10 



S.D. 

.08 

.08 

.07 

.09 

.04 



Geometry Items 
Mean S.D. 



.26 
.15 
.24 
.27 
.27 



.11 
.08 
.15 
.15 
.17 



6 
7 
8 
9 
10 



20 
13 
10 
15 
15 



.15 
.15 
,17 
.16 
.19 



.05 
.06 
.07 
.08 
.08 



.12 
.14 
.16 
.11 
.14 



.07 
.11 
.07 
.05 
.08 



.20 
.19 
.17 

.24 
.24 



.12 
.10 
.12 
.16 
.16 



11 

12 
13 
14 

15 



15 
14, 
16 
14 
25 



.23 
.25 
.16 
.27 
.25 



.11 
.11 
.07 
.09 
.08 



.18 
.22 
.15 
.23 
.25 



.08 
.12 
.07 
.13 
.10 



.25 
.27 
.18 
.24 
.23 



.16 
.11 
.09 
.13 
.17 



Table 11 

Weighted "Standardized Mean Residuals of Within School 
Item Difficulties by Content Category for 
Schools Within Census Division 6 
Book 4, 13 year olds, 1978 



Content Category 1 


2 


3 


4 


5 


6 


7 


ocnooi 
8 


9 


10 


11 


12 


13 


Story Problems 


.50 


- .99 


.35 


.02 


- .72 


.29 


.71 


- .58 


- .21 


- .30 


.71 


1.13 


1.03 


Calculation 


- .33 


.27 


-1.02 


- .77 


-1.58 


.73 


.24 


1.10 


.48 


.78 


-1.21 


.11 


1.10 


Geometry — 
Definition 


-1.52 


- .40 


1.17 


1.74 


1.15 


■ 

.16 


- .52 


.77 


1.90 


l.Ol 


.43 


- .90 


.00 


Geometry — 
Application 


.18 


.60 


.12 


.85 


- .64 


.74 


.94 


1.41 


1.00 


1.42 


1.45 


2.03 


.89 


Graphs 


- .03 


.77 


.60 


.19 


.55 


- .43 


.65 


.07 


- .01 


-1.00 


- .32 


- .21 . 


-1.91 


Estimation 


2.12 


.42 


1.26 


.43 


-1.25 


- .8 .0 


- .97 


2.00 


1.28 


- .27 


- .57 


1.34 


.12. 


Sign Numbers 


-1.66 


3.88 


-2.45 


-2.23 


1.26 


.55 


- .01 


-1.07 


-1.04 


-1.24 


.15 


- .57 


.39 


Fractions 


.85 


- .99 


-1.45 


■ .:79 


1.68 


-1 . 52, 


.83 


.03 ' 


-2.44 


2.70 


- .03 


.75 


.86 


Blocks 


.41 


.74 


-1.22 


.27 


1.19 


1.36 , 


.80 


- .49 


- .76 


-3.18 


- .08 


.06 


. 1.35 


Unclassified 


- .41 


-1.25 


1.11 


-1.40 


- .78 


- .53 


-1.88 


-1.58 


-.84 


-1.01 


- .60 


-1.85 


-2.48 



. Table 12 - 
Weighted Standardized Mean Residuals of Within School 
Item Difficulties by Content Category 
Schools Within Census Division 7 
Book A, 13 year olds, 1978 





2 


3 


A 


5 


- 

6 


School 
7 8 


9 


10 


1 1 


i z 


1 J 


1 L 

JL *T 


15 


Story Problems 


L m {J J 


.19 


1.18 


.23 


-1.03 


1.A9 


- .A2 


- .11 


- .46 


• i J 




— . 1 J 




- .05 


- .16 


Calculation 


Q Q 
• OO 


1.05 


1.35 


.10 


- .AO 


- .89 


.68 


- .08 


1 . J J 


i « 0 i 




1.45 


- .45 


1.36 


.70 


Geometry 
Definition 


.67 


- .05 


- .18 


- .79 


.80 


- .13 


-2.08 


1.56 


- .87 


-1.68 


-2.67 


.25 


-1.07 


- .32 


-1.50 


Geometry 
Application 


.55 


3.27 


- .73 


-1.20 


-1.76 


- .50 


.88 


- .A2 


.13 


.A6 


- .38 


2.03 


- .74 


2.46 


.32 V 


Graphs 


-1.31 


-1.91 


1.23 


1.33 


.39 


1.09 


.81 


1.32 


1.17 


- .25 


-3.43 


- .67 


.90 


-2.87 


.01 


Estimation 


1.A6 


.89 


- .31 


.78 


-1.21 


.06 


3.59 


-1.15 


.80 


.33 


- .00 


.69 


- .55 


1.35 


- .86 


Sign Numbers 


.59 


-1.5A 


.12 


- .58 


2.85 


-2.55 


-1.9A 


- .38 


-2.A2 


-2.5A 


-2.A7 


- .76 


3.35 


- .57 


2.44 


Fractions 


1.03 


1.10 


l.AO 


1.58 


- .91 


.33 


- .Al 


- .6A 


1.7A 


1.01 


-2.80 


-1.07 


- .56 


-1.81 


.45 


Blocks 


- .73 


.26 


1.19 


.19 


- .75 


- .Al 


.51 


.1.05 


1.06 


1.47 


-1.80 


.09 


- .05 


.50 


.24 


Unclassified 


-1.19 


-2.97 


-1.93 


- .98 


1.81 


.71 


- .A7 


- .87 


-1.89 


.47 


-3.9A 


-1.70 


.10 


- .20 


- .67 



ERJC 



Stem 



Table 13 

Stem and Leaf Display of p-Values for Cognitive^ Items- 
Book 5, 13 year olds, 1978 
(N = 63) 



Tilf ~ZZ_ Frequency 



5.17 5. OA 



8.18 7.29 5.5A 



A.A5 1.12 l.OA 0.15 
9.70 6.A5 



8.69 , 7.15 7.07 6.86 



8.91 7.A9 6.96 . 5.95 
2.33 



2 



A. 80 0.78 - 2 



3 



A.2A 3.75 2.78 2.66 2.37 1.A8 6 
8.59 6.AA 6.28 3 



A 
2 

A. 09 2.27 0.A8 ''3 



A 



A.A7 3.61 3.33 2.52 2.11 1.06 6 
8.86 8.25 7.97 5.9A ^ 



3.3vQ 2.65 2. AO 2. AO 0.78 5 

4 
1 
2 



9.85 5.59 ~ 

A. 21 2.^22 1.20 , 3 

\ ■ '\ ■ 

9.22 8.81. . ^' ^ 

3.28 ' ' . . ^ 

9.91 9.A6 _ 9.26 8.90 A 

2.88 1.62 ^ 

63 




Table 14 .■' 
Means, Standard Deviations »^ Skewness, and the Pearson Product Moment 
Correlations Among Sato's Caution Index, Modified Caution Index and the 
Total rest Scoire for the Total Set and Two Subsets of Iteras 
^Book 5, 13 year olds, 1978 
'(N = 1231) 












Variable 










VariaDie 


i 


9 
L 


o • 


4 


5 


D 


7 


fl 


9 


^■ 

All Items 




















1. Sato's Caution Index 


1.00 


.99 


-.21 


.66 


.66 


-.23 


.46 


.US 


-.1,8 


2. Modified Caution Index 


I 


1.00 


-.15 


.66 


.66 


-.17 


.48 


.48 


-.13 


3. Total Test Score 






1.00 


-.02 


-.04 


.96 


•12 


•1^ 


.64 


Arithmetic Items 


• 


















A. Sato!s Caution Index' 








1 .00 


.99 


-.04 


.02 


.03 


.00 


5. Modified Caution Indes 










1.00 


-.05 


.02 


.02 


-.01 


6. Total Test Score 

^ 












1.00 


.11 


.13 


.50 


Geometry Itenxs 








s 












7. Sato's Caution Index 






** 

. 1 








1.00 


.99 


-.02 


8. Modified Caution Index 
















1.00 


.01 


.9. Total Test Score 


















1.00 


Mean * 




.16 


33.74 


1 '^'^ 


.18 


20.26 


.21 


.10 


5.72 


Standard Deviation ^ * ' 


.12 


.06 


10.61 


\ -19 


.09 


7.20 


.26 


.13 


1.68 


Skevmess* 


.93 


.80 


-.li 


1.22 


1.21 


-.23 


1.86 


1.93 


-.02 



; 



Table 15 

Pearson Product Moment Correlatibna* of School, Student Background, 

and Affective Domain Variables with Modified Caution Index and the 

Total Test "Score for the Total Set and Two Subsets of Items 
» 

^ Book 5, 13 year olds, 1978 

(N = 1231) 





All Items 


Arithmetic 


Items 


Geometry 


Items 




Modified 


Total 


Modified 


Total 


Modified 


Total 


Variable 


Caution 


Score 


Caution 


Score 


Caution 


Scox."e 


1. Percent of Coiomunity 
iin^lcilled 


-.02 


-.06 


-.02 


-.05 


-.01 H 


-.06 


?- pprrpnt of community 
not employed * 


.03 


-.21 


.01 


-.19 


-.04 


-.15 


3. Percent of community 
on welfatre 


.07 


-.26 


OA 


-.25 ' 


-.04 


-.14 • 


4» Percent of community 
farra workers 


-.03 


-.01 


-.02 


.00' } 


.] -.03 


-.02 


5. Percent of community black 


.03 


-.36 


.05 








6. Percent of community 
1 protessipnai workers 


.02 


.22 - 


-.01 




.07 


..16 


7. Percent of community 
skilled workers . 


-.03 


.09 


.02 


.09 _ 


.00 


.07 


8. Number of students quali- 
fied to receive Title I 


, .05 


-.23 


.04 


-.22 


/ -.06 


-.12 


9. School enrollment size 


.02 


-.12 


-.01 


-.12 


,02 


-.09 


10. Sex^ 


-.16 


.01 


-.11 


.05 


-.08 


-.05 ^ 


b I 

11. Family owns calculator 


-.06 


.20 


-.01 


.20 


.02 


.12 


c 

12. Type of school 


-.03 


.15 


.02 


.14 


.02 


.11 


13. Like mathematics scale 


* .00 


' .06 


.04 


.05 


.01 


.08 


14. Importance of mathematics 

scale 


-.06/ 


.15 


-.02 


.14 


.02 . 


.15 


15. Easiness of mathematics 
scale 


.02 


.31, 


^.05 ' 


.29 


.09 


.20 

c 



^Sex is coded 1 for* males and 2 -for females. 
^Family owns calculator is coded 0 for no and 1 for yes. 
^Type of school is coded .1 for public and 2 for private Catholic. 
★Correlations greater in absolute magnitude than .07 are significant at 



p < .01 









\ 

Table 16 




1 






Analysis 


of Variance Summary Results 


for the Modified Caution Index 




with the Test 


Score 


as a Covariate for the Total Set 


and Two 


Subsets of 


Items 






Book 


5, 13 year 


olds, 1978 












(N 


= 1231) 














All Items 


Arithmetic 


I terns 


Geometry 


Items 






Mecin 




Mean 




Mean 




Sourpe 


df 


Square 


F 


Square 


F 


Square 


F 


ANOVA 
















Main Effect 

Community size 
Residual 


3 

1227 


.005 
.004 


1.30 


.005 
.009 


.53 


.048 
.017 


2.89* 


ANCOVA 

Covariate 


1 


. .095 


26.43** 


.043 




.uuz \ 


.11 


Main Effect 

Community Size 
Residual 


3 

1226 


.005 
.004 


1.49 


.002 
.009 


.29 


.047 
.017 


2.85* 


ANOVA 
















Main Effect 

Race-Ethnicity 
Residual 


3 

1227 


.015 
.004 


4.19** 

\ 

I 


. .016 
.009 


1.82 


.066 
.016 


3.99** 


ANCOVA 

Covariate 


1 


.095 


26.43** 


.043 


A Q 




.11 


Main Effect 

Race-Ethnicity 
Residual 


3 

1226 


.005 
.004 


1.48 


.009 
.009 


1.03 


.065 


3.96** 


ANOVA 
















-Main Effect 
Grade Level 
Residual 


4 

1226 


.011 " 
.004 


3.04* 


.011 
.009 


1.33 


.008 
.017 


.51 


ANCOVA 

Covariate 


1 


.095 


26.44** 


.043 




. KJKJZ. 


.11 


Main Effect 

Grade Level 
Residual 


\ 4 
1225 


.005 
.004 


1.48 


.009 


1.32 


.008 
.017 


.49 


ANOVA 
















Main Effect 

Censu^Divlslon 8.. 
' Residual 1222 


.001 
.004 


.35 


.001 
.009 


.13 


.037 

,016 

/ 


2.24* 


ANCOVA 

Covariate 


1 


.095 


26.38** 


.043 


4.93* 


.002 


.11 


Main Effect 

Census Division 8 
Residual 1221 


.003 
.004 


.87 


.002 
.009 


.22 


.037 
.016 


2,24* 


*p < ,05 
















^vVp < .01 

O 

mc 








fe'a— 









Table 17 j 
Percentages of Varlajflon for the Components In 
Mathematics Achievement and on the Modified Caution Index 
Book 5, 13 year olds, 1978 / 



All items ! 
Arithmetic it^tns 
Geometry items 

Modified Caution Index 
All items 
Arithmetic Items 
^ Geometry items 







yfeetween Census 
/ Division 


School Within 
Censusj Divisions Error 


Mathematics A 


ihievement 


/ 

/ 


i 



5.19 
4.31 
2.01 



23.32 
22.49 
9.78 



3.51 
2.66 
2.33 



71.49 
73.49 
88.21 



96.49 
97.34 
96.55 



Table 18 

Summary of F Ratios for Hierarchical Analyses of Variance for the 
Modified Caution Index on the Total Set and Two Subsets of Items 









Book 5, 13 year 


olds, 1978 






Effect 








All 
Items 


Arithmetic 
Items 


Geometry 
Items 


Census Division 








.57 


.33 


4.37** 


Schools within Census Division 




1.56** 


1.44** 


1.39* 


Schools 


within 


Census 


Division 


1 


1.51 


1.52 


3.92** 


Schools 


within 


Census 


Division 


2 


1.78* 


1.21 


1.15 


Schools 


within 


Census 


Division 


3 


1 CO 

1 . D J 


1 AT 


1 7S* 


Schools 


within 


Census 


•Division 


A 


1.39 


1.18 


2.01 

1 


Schools 


within 


Census 


Division 


5 


.92 


1.02 


.37 


Schools 


within 


Census 


Division 


6 


1.07 


.76 


.67 


Schools 


wi thin 


Census 


Division 


7 


1.13 


1.76 


.84 


Schools 


within 


Census 


Division 


8 


1.21 


.88 


2.11 


Schools 


within 


Census 


Division 


9 


2.71** 


2.34** 


1.27 



*p < .01 
**p < .001 



ERIC 
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Means and Standard Deviations on Modified Caution Indices on the 



Table 19 

Means and Standard Deviations on Modified Caution Indices on the 
Total Set and Two Subsets^ of Items for 
Schools Within Census Dijvision 1 
Book 5, 13 year oldsj 1978 

School Sample All Items Arithmetic Items Geometry Items 

Numb er Size Mean S.D. Mean S.D. Mean S.D. 

1 15 .15 .04 .11 .08 .15 ,07 

2 14 .19 .07 .21 .07 .25 .25 

3 12 .16 .06 .15 .08 .11 .12 

4 13 .14 .06 .17 .07 .08 .14 
^5 20 .17 .07 .19 .08 .15 .23 

6 15 .14 .06 .17 .07 .03 .06 

7 14 .15 .05 .18 .09 .14 .15 

8 13 .16 .05 .16 .08 .09 .09 



ERiC 
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Table 20. 

Means and Standard Deviations on Modified Caution Indices 







on the Total 


Set and 


Two Subsets of Items 


for 








Schools 


Within 


Census Division 9 










Book 


5, 1-3 


year olds. 


1978 






School 


Sample 


All Items 


Arithmetic Items 


Geometry 


Items 


Number 


Size 


Mean 


S.D. 


Mean 


S.D. 


Mean 


SiD. 


1 


18 


.16 


.04 


.16 


.06 


.12 


.11 


2 


13 


.19 


.04 


.23 


.06 


.09 


.08 


3 


19 


.18 


.06 


.20 


.10 


.04 


.15 


4 


17 


.15 


.04 


.16 


.07 


.09 


.09 


5 


23 


.14 


.05 


.17 


.09 . 


.08 


.11 


6 


15 


.16 


.04 


.17 


.11 


.14 


.14 


7 


17 


.18 


.08 


.20 


.19 


.15 


.10 


8 


18 


.16 


.06 


.18 


.08 


.06 


.09 


9 


17 


.19 


.05 


.11 


.07 


.09 


.13 


10 


17 


.19 


.05 


.19 


.09 


.13 


.12 


11 


12 


.14' 


.05 
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Table 21 

Weighted Standardized Mean Residuals of 
Within School Item Difficulties by Content Category for Schools 
Within Census Division 1 
Book 5, 13 year olds, 1978 



School 



Content Category 


1 


2 


3 


4 


5 


6 


7 


8 


Story Problems 


- .03 


-2.10 


.50 


-2.73 


- .61 


- .46 


- a6 


.65 


Calculations 


- .49 


- .16 


- .78 


.30 


.8" 


1.69 


.47 


.86 


Geometry 


-1.32 


y<22 


- .34 


-1.03 


- .03 


- .23 


- .04 


3.16 


Metric 


.68 j 


' 1.72 


-1.21 


2.76 


- .29 


- .53 


- .19 


-2.50 


Graphs 


-1.3« 


- .09 


.64 


.45 


- .69 


- .97 


.03 


.01 


Definitions 


1.51 


-1.18 


1.00 


.32 


1.61 


,63 


-1.89 


- .59 


Numeration 


-1.59 


2.97 


-2.11 


.89 


1.24 


-1.24 


.07 


- .98 


Fraction to Decimals 


2.81 


- .85 


.24 


- .06 


-3,09 


1.70 


.61 


-3.35 


Unclassified 


.97 


1.81 


1.89 


1.21 


1.34 


-1.85 


1.53 


- .26 
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Table 22 

Weighted St-Rndardized Mean Residuals of Within School Item Difficulties by Content Category for 

Schools Within Census Division 9 
Book 5, 13 year olds, 1978 



Content Category 



School Story Calcu- Geometry Metric 



Graphs 



Definition Numeration Fractions Unclassified 



iniber 


Problems 


latlons 












to Decimals 




1 


1.06 


.59 


-1.01 


- .26 


.80 


.91 


-1.27 


-2.55 


.96 


2 
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.83 


1 r» 1 
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1 . 00 
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A A 
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Figure 1. Scatter dia'gram of modified caution index with total math 

test score. (A 9 denotes 9 or more observations, otherwise 
the numeral corresponds to the number of obsorvations; 
Book 4, 13 year olds, 1978.) 
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Figure 2. Scatter diagram of modified caution index with total arith 
metic test- score. (A 9. denotes 9. or more observations, 
otherwise the numeral cortesponds to the number of observe 
tions; Book 4, 13 year olds, 1978.) « 
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Figure 3. Scatter diagram of modified caution index with total 

geometry test score. (A 9 denotes 9 or more observations 
otherwise the numeral corresponds to the number of obser- 
vations; Book 4, 13 year olds, 1978.) • 
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Figure 4. Scatter diagram of modified caution index from the arithmeti 
test with modified caution index from the geometry test. 
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Box and whisker plot of modified caution Index for whites and blacks with the 
total math test score interval split into deciles. (Book 4, 13 year olds. 1978 
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Figure 6, 



Box and whisker plot of modified caution index for whites and tblacks with the total 
arithmetic test score interval split into deciles. (Book 4, 13 year olds, 1978. X 
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Figure 7, Box and whisker plot of mpdified caution index for whites and blacks with the ^ 
total geometry test score inteirval split into quintiles. (Book 4, 13 year olds, O/ 
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Figure 8* 



Box and whisker plot of modified caution index tor males and females with the total 
math test score interval split into deciles, (Book 4, 13 year olds, 1978.) 
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Figure 9 



Box and whisker plot of modified caution index for males and females with the total 
arithmetic test score interval split into deciles. (Book 4, 13 year olds, 1978.) 
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Figure 10, Box and whisker plot of modified caution index for nfales iind females vwith the 

total geometry test score interval split into quantiles, (Book 4, 13 year olds, 
1978:) , 
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Figure 11. 



Total Math Test Performance Level 

Box and whisker plot of modified caution Index for students with low and high 
scores on the usefulness of mathematics scale with the total math test score 
interval split Intov deciles. ' (Book 4, 13 year olds, 19780 
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Figure 13. Box and whisker plot of modified caution index for students with low and high scores 
on the usefulness of mathematics scale with the total geometry test score interval 
split into quintiles. (Book A, 13 year aids, 1978 -) 



.350 



ERIC 



(U 



o 

•H 

u 

CO 

a 

QJ 
•H 

. O 



.233 



.117 



.000 



.117 



.233 



-.3bC 

WlflTE DEC 
DLACK DEC 



LES 



1 
I 
I 

1+ 

I 
1 
! 

+H- 
J 
I 
\ 

' ! 
1 
I 
I 
1 
I 
I 



Fieure lA. 



*+* 1 



I ! 
1 I 
1 1 
1+1 

1 1 
t 1 
+- + 
1 



I I 
t ! 
*+* 

I t 
1 I 
I 1 
1 i 
+-+ 



I 



I 



I 1 
1 1 
I+l 
*.* 

1 1 
I I 



1 1 
1 1 
*+* 
1 1 
1 1 



0 
0 



I 1 

I 1 

I 1 

1 1 



1 1 
1+1 
*•* 

I 1 
I 1 
+-+ 



+-+ 
1 1 
I 1 
*+* 

I -1 
1 1 

-H-+ 



1 
I 

i-+ 
1 1 
I I 
*-* 

I+l 
I 1 
I 1 



+-+ +-+ 
ill! 



1 I 
*+* 

t i 



1 I 
I I 
1 1 
+-+ 

I 

I 

I 



1 1 
1 I 
*+* 
1 1 
+-+ 

1 

I 

1 

1 

0 
0 
0 



+-+ 
I 1 
I+I 

1 1 

-H-+ 
I 



+-+ 
1 I 
*+* 

1 I 
1 1 
+H- 

1 

I 

1 

1 

0 



+-+ -H-+ 
1111 



I 1 
1 1 
1 1 



I I 
*+* 
1 ! 
+-+ 



! I 
1 1 
1 I 
I+l 

( i 

I 1 
I I 
+-+ 
1 



1 



\%t 
I+l 
*•* 

1 1 
1 1 

+-H 
I 
1 
I 
I 
I 
I 

0 



+-+ 
*+* 



iO'J 



2 3 4 5 6 7 8 9 10 

1-2 3 4 5 6 7 8 9 , 10 

Total Math Test Performance Level 
Box and whisker plot of modified caution index residuals for whites and blacks 
with the total math test score interval split into deciles'. (Book A, 13 year olds, 
1978.) 



10 



1 



.350 



.233 



.117 



(U 

M .000 



-.117 



-.233 



-.350 
WH1.«L DEC 
BLACK DEC 



-fH- 
I I 
I I 

I I 
I 1 
I 1 
+-+ 



+-+ 
I 1 
t I 

t ft 
t 1 



+-+ 
I I 
*.* 

I+! 
! I 
I 1 



* 

0 

0 
0 

I 
I 
I 
I 
I 
I 
I 
I 

+-+ 
I 
I 
I 

1+ 

I 

! 
I 

I 
I 
! 
I 
I 
I 
I 
I 



I 
I 
t 

*+* 

I 
I 
I 



+-+ 
I I 

I : 
*+* 

I I 
t t 
I 1 
+-+ 



r 
I 

+-+ 
I I 
I I 
i I 
*+* 

I I 
1 I 
+-+ 

I 

I 

I 

I 

































• 




V 


















* 




* 








0 




















0 
















* 




0 






1 


A 




0 








0 




* 


I 


0 




0 




0 




! 




0 


I 


0 








0 








0 




0 




j 












n 

M 












[ 




\ 


1 


0 


I 


1 




1 




\ 


1 


I 


1 


1 




J 




I 






I 

-f-+ 


+-+ 


I 


I 
1 


I ! 

Tl 






+-+ 




+-+ 


I 1 


1 1 


i 1 


1 








1 i 


{ [ 


I I 


I 1 


1 I 


1 1 




M 


+— H 




I 1 




t I 


I+I 


1 i 


*-* 


I 1 


1 1 


I 1 




i+i 




l+I 




i+i 


l+i 


!+I 


A.* 


I+I 




t ! 




*-* 


1 1 




1 1 






*-* 


+-+ 
1 i 


*-* 

1 i 




I I 
I t 


I 1 
+-+ 


I i 


1 1 


1 t 
+-+ 




1 i 
1 1 


*+* 


+-+ 




+-+ 


I 










+-+ 


1 i 


I 


+-+ 




i 
1 












+-+ 
1 
1 


.1 

! 
1 





LES 



4 5 ^ 6 ' 7 8 

3 4 5 6 7 8 

Total Arithmetic Test Performance Level 



10 



10 



Figure 15. 



10 



2 



Box and whisker plot of modified caution index residuals for whites and blacks 
with the total arithmetic test ccore interval split^into deciles. (Book 4, 
13 year olds, 1978.) . * 
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Figure 21. 
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Total Math Test Performance Level 

Box and whisker plot of modified caution index for males and females with the 
total math test score interval split into deciles- (Book 5, 13* year olds, 1978.) 
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Figure 22. 
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Total Arithmetic Test Performance Level 

Box and whisker plot of modified caution index for males and females with 
total arithmetic test score interval split into deciles. (Book 5, 13 vcar 
1978.) 
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Appendix A 

Classification, Percentage Correct, Sato's Caution Index, Modified 

Caution Index and the Point-Biserial for Cognitive Items 

Book 4, 13 year olds, 1978 
(N - 68) 



Item 


Item 




Percentage 


Sato s Caution 


Modified 


Point*" 


Nuraber^ 


Classification* 


Correct 


Index 


CautioTi Index 


Biserial 


2 


M 




O 1 1 / 

83.34 


.45 


.21 


o /. 

« J4 


3 


A 




71 •69 


.36 


. 1 / 


• ^6 


4A 


A 




50.96 


.24 


.12 


• 61 


4B 


A 




63 .97 


.32 


. .15 


• 5J • 


5A 


A 




97.17 


.52 


.24 


• 17 


5B 


A 




C%/. C /• 

9^ .5^ 


/ Q 

.49 


. / J 


o o 
• 11 


5C 


A 




Q / 1 ^ 

o4 .lo 


.56 




• 2/ 


6A 






Q /• no 


/. Q 


o o 
.22 


• j2 


6B 


G 




23 .43 


.54 


O Q 

. 2o 


• 35 


" 6C " 


G 




34.39 


.46 


.24 


• 43 


6D 


G 




1 O / C 

19 .45 


/. Q 

• Hy 


. 2o 


• Jo 


6E 


G 




/CO/ 

45.o4 


/. c 

• HD 


. 2 J 


A ^ 
• H J 


7 


A 




2J .14 


/ O 


.22 


A/. 
• HH 


8 


M. 




39.68 


• 52 


. 2q 


• jy 


9 


A 




44.36 


.42 


.21 


.47 


lOA 


G 




65.78 


• 7 7 


.Jo 


1 O 

• lo 


lOB 


G 




57 .12 


.65 


. J2 


• 2o 


IOC 


G 




31.68 


1 .08 


.5o 


-•06 


lOD 


G 




88.14 


.55 


.26 


.25 


lOE 


G 




50.55 


1.04 


.52 


-;o3 


llA 


T 




85.02 


.55 


.25 


• 27 


IIB 


T 




53.10 


.34 


.17 


• 53 


lie 


T 




79.85 


.28 


.13 


• 48 


12 


A 




24.33 


.,.55 


.29 


.35 


13 


A 




57.00 


.79 


.39 


.17 


14A 


A 




32.87 


.46 


.24 


.43 


14B 


A 




38.65 


.42 


.21 


.47 


l^.C 


A 




28.48 


.42 


.22 . 


.45 


14D 


A 




30.57 


.40 


.21 


.48 


15 


A 




64.87 


.32 


.15 


.52 


16 


A 




65.12 


.35 


.17 


.50 


17A 


' G 




54.37 


.41 


.20 


.47 


17B 


G 




31.23 


.54 


.28 


.37 


17C 


G 




35.13 ' 


.63 


.32 


5j .30 


18A 


G 




46.37 


.70 


.35 


.24 



V 
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Appendix A (continued) 



Item 
Number 

18B 
18C 
19 
20 
21A 

21B 
21C 
22 
23A 
23B 

23C 
24 
, 25A 
25B 
25C 

26 

27A 
27B 

28A 
28B 

29A 
298 
29C 
30 
31 

32 
33 
34 
35 
36 

37 

3V\^ 



Percentage Sar.o's Cautioa Modified 



Classification* 


Correct 


Index 


Caution Index 


G 


17.28 


.60 


• 31 


G 


16.50 


.50 


.27 


A 


25.24 


.58 


•30 


A 


OO . O J 


.53 


• 25 


A 


' 7.35 


.77 


.40 


A 


^ 24.29 


.66 


• 35 


A - 


19.45 


.82 




A 


32.50 


.31 


• 16 


A 


Q1 10' 


.36 


• 17 


A 


79.44 


.46 


• 22 


A 


82.81 


.43 


.20 


G 


29.13 


.62 


• 32 


A 


68.24 


.58 


.28 


A - 


25.77 


.40 


.21 


A 


28.44 


.39 


.20 " 


G 


3.90 


.22 . 


.IZ 


A 


22.65 


.40 


• 21 


A 


43.33 


^44 


.22 


M 


ftS 90 


.49 


.24 


M 


71.48 


.51 


.25 


T 


79.48 


.31 


.14 


' T 


86.62 


.27 


.13 


T 


35.70 


.68 


.35 


A 


1 A 7R 


.97 


.52 


G~ 


50.14 


.59 


.30 


A 


33.28 


.37 


.19 


A 


2.05 


.31 


.17 


G 


47.97 


.32 


.16 


A 


33.03 


.42 


.22 


T 


55.31 


.52 


.26 


A 


49.65 


.37 


.19 



Point- 
Blserlal 



A 

M- 



26.14 
54.78 



.36 
.47 



.19 

.23 



.29 
.35 
.32 
.27 
.13 

.26 
.13 
.55 
.33 
.36 

.36 

.30 
.32 
.47 

T48 

.35 
.46 
.45 
.39 
.36 

.46 
.43 
.26 
.02 
.33 

.51 
.24 
.55 
.46 ' 
.39 

.51 
.50 

.42 ' 



iitem cla>s^iflcation codes are A for arithmetic, G for geometry, T for tables 
and graphs for miscellaneous. 
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Appendix B 

Means, Sl^andard Deviations, and Pearson Product Moment 



Correlations on 


Four Item 


Statistics 




Book 4, 13 


year olds , 


1978 




(N 


= 68) 


0 




Variable 

. " - 


Variable 1 


2 


3 


4 


1, Percentage. Correct „ 1*00 


.02 


- .15 


- .26 


2. Point-Biserial 


1.00 


- .88 


- .36 


3, Sato's Caution Index 




1.00 


.99 


4. Modified Caution Index 






1.00 


Mean 48.43 


• 36 


.50 


.25 


'standard Deviation 25.08 


.14 


.18 


.09 
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Appendix C 



Classification, ^ercenta&e Correct, Sato's Caution Index, Modified 
Caution Index and tvie Point-Bi^eriaP for Cognitive Items 
Book 5, 13 year olds, 1978 
(N - 63) 



Item Item Percentage 

Number Classification* Correct 



Sato's Caution Modified Point- 
Index Caution Index' Biserial 



*> 

z 




A 


fl3 7 S 


3 




A 


' • ^ 






- A --^ — ' 
A 


94.80 






A 




5A 




G 


88.18 






n 


82.37 


5c 




G 


lf% LL 

/ D . HH 


5D 




G. 


O / . >1 7 


6A 




rX 


S7 07 


6B 




M 


53.33 


oC 




M 


47 97 


7 A 




T 
1 


69 70 


7B 




T 
i 




8A 




A 

A 


18-91 


8B 




A 


35.95 


oC 




A 


42 .40 


9 




A 


13.28 


lOA 




A 

A 


95.17 


iuB 




A • 

A 


95.04 






A 


90.78 


IK 




T 


42.40 


12 




A 


36.96 


13 




A 


22.22 


14 




A 


74.45 


15A 




A 


64.09 


15B 




A 


70.15 


15C 




, A 


76.28 


16 




A 


. 19.21 


17 




A 


24.21 


18 




A 


53.69 


19A 




T 


40.78 


19B 




T 


51.06 


19S 




T 


42.65 


20A 




M 


1.62 


20B 




M 


2.88 



.52 
.28 
.51 
.53 
.51 

.81 
.93 
.54 
.52 
.39 

.83 
.42 
.43 
.36 
.35 

.49 
1.17 
.41 
.41 
.41 

.34 
.30 
.39 
.43 
.31 

.34 
.27 
.80 
.29 
.40 

.40 
.33 
.36 
.33 
.27 



.26 
.i4 " 
.27 
.27 
.26 

.41 
.46 
.27 
.26 
.20 

.41 
.21 
.21 
.18 
.17 

.25 
.58 
.21 
.21 
.21 

.17 
.15 
.19 
.22 
.16 

.17 
.14 
.40 
.14 
.20 

.20 
.16 
.18 
.16 
.13 



.32 
.59 
.22 
.33 
.29 

.13 
.05 
.29 
.40 
.50 

.14 
.45 
.A5 
.'52 
.53 

.42 
-.10 
.26 
.26 
.33 

.54 
.57 
.44 
.42 
.55 

.51 
.54 
.14 
.52 
.49 

.49 
.56 
.52 
.18 
.25 
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•Appendix C (continued) 



Item 



Item 



Number 


Classlf :lc 


21 


n 
\j 


22 


\j. 


23A 


A 


23B 


A 


23C 






CI 


Z!) 


CI 


0^ 
/o 


A 
A 


27 


A 


28 


M- 


Z9 


1 


iOA 


A 

A 


3 OB 


A 


30C _ 


A 


30D 


A 


JUE 


A 


31 


n 


32 


c 


33A 


M 


33B 


M 


34 


A 


35 


M 


36 


A 


37 


A 


38 


A 


39 


G 


40 


A 


41 


• M 



Percentage 
Correct 


Sato's Caution 
Index 


Modified 
Caution Index 


Point- 
Biserial 


82.66 


.60 


.30^ 


.27 


25.59 


.40 


.20 




56.86 


.57 


.29 


.3f 


48.86 


.49 " 


.25 


.42 


48.25 


.43 


.22 


.47 


62.27 


.65 


. .33 


.28 


18.81 


.83 


.41 


.11 


, 54 47 


.63 


-.32 


.30 


53.61 


.39 


.19 


.51, 


8.90 


1.08 


.53 


-.04 


71.04 


.45 


.22 


:43 


60.48 


.16 


.08 


.68 


57.15 


.18 


.09 


.67 


52. U 


.17 


_.08 


.69 


52.52 


.23 " 


.12 


.64 


37.49 


.22 


.11 


.63 


9.46 


.22 


.11 


.42 


9.91 


.63 


.31 


.21 


82.78 


.58 


.29 


.29 


29.85 


.90 


.45 


' . .08 


71 .12 


.66 


.33 


.26 


85.54 


.28 


.14 


.46 


84.24 


.51 


.26 


.32 


32.33 


. ■ .34 


.17 


.52 


81.48 


.47 


.24 


.36 


21.20 


.45 


.22 


.39 


45.94 


.46 


.23 


.45 


9.26 


.59 


.29 


.22 



*Itera classification codes are A -for arithmetic, G for Geometry, T for tables 
and graphs and M for miscellaneous. - 



Appendix D 



Means, Standard Deviations, and Pearson Product Mpraeut 
Correlations on Four Item Statistics 
Book .5, 13 year olds, 1978 
(N - 63) 



Variable 



Variable 
2 3 



I. 


^ — 

Percentage Correct 


1.00 . * 


.09 


- .09 


- .06 




Point-Bis e rial 


: 


1.00 


^ - .89 


^.89L^ 


2^ 


9 




3. 


Sato's Caution Index 






1.00 


.99 


4. 


Modified Caution Index 








1.00 




Mean 


52.74 


.38 


.48 


.24 




Standard Deviation 


26.13 


.18 


.21 


.11 
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